The R scripts in this repository reproduce all results from Pietryka & Macintosh, "ANES Scales Often Don't Measure What You Think They Measure."  The Journal of Politics.

For any questions, contact Matt Pietryka (matthew.pietryka@gmail.com)


1. Directory Structure -----------------------------------------------

The replication files assume the following directory structure:


├── DataClean
├── Functions
├── Analysis
│   └── report-results
│       ├── Authoritarianism
│       ├── Figures
│       └── Tables
├── Data
│   ├── Derived
│   │   ├── 2012
│   │   └── 2016
│   └── Source
└── README.md

To replicate these analyses, begin by creating this structure and setting the R working directory to the top level. Or modify all R scripts to correspond to the structure on your machine. 

Note: The Functions directory contains R functions that are used in the scripts contained in DataClean and Analysis directories


2. Obtaining the data ----------------------------------------------------


Before running the scripts, the data must be acquired from the ANES.
   * 2012 ANES
     - We used the version dated 05/04/2016
     - The current version are here:
       https://electionstudies.org/data-center/2012-time-series-study/
   * 2016 ANES
     - We used the version dated 5/2/2017
     - The current version are here:
       https://electionstudies.org/data-center/2016-time-series-study/

To produce taylor series standard errors with the 2016 data, one must also obtain the PSU variable from the restricted access data file:
https://electionstudies.org/restricted-data-access/

The scripts assume the data obtained from the ANES are located in the Data/Source subdirectory.

2. Cleaning the data -----------------------------------------------

The scripts in DataClean.zip process the raw ANES data to prepare them for the analyses.

The DataClean.zip contains the following scripts, which should be run in order:

  1. anes-1-subset_restricted_data: This script extracts the PSU and ID columns needed to estimate taylor series standard errors. Skip this script if you do not have access to the restricted data.
  2. anes-2-define_groupvars: Clean the grouping variables
  3. anes-3-rename_scalevars: Renames the 2016 scale variables so they match the 2012 names
  4. anes-4-define_scales: Clean the scale items and create the scales
  X. scale_items: this is a helper file that is used in several other scripts, but is not run on its own.


3. Performming the analysis --------------------------------------

The scripts in Analysis.zip execute all analyses reported in the manuscript and supporting information.

The Analysis.zip contains two versions of each script, prefixed AP12 or AP16, for analysis of the 2012 or 2016 data. For each year, execute the following scripts in order:

  1. AP1[2/6]-1-Load_Data: Load the data and create the nested data object used for the analysis
  2. AP1[2/6]-2-Rasch_Models: Fit the Rasch models for each scale x grouping variable combination
  3. AP1[2/6]-3-DIF: Fit the ANOVAs used to evaluate DIF
  4. AP1[2/6]-4-Compare_New_and_Old: Compare group differences when using the off-the-shelf vs DIF-resolved scales

After running these scripts, one can use the scripts in the report-results subdirectory to reproduce all plots and tables in the manuscript or online supporting information. Note that reproducing the authoritarianism-specific plots (Figure 6 and Online SI G) requires running all scripts in the Authoritarianism subdirectory in sequential order.


4. Software details ----------------------------------------

Results may differ using different versions of the R packages or ANES data.

All scripts were run using a 64-bit Windows 10 machine with R version 3.3.3 (2017-03-06). Contact Pietryka (matthew.pietryka@gmail.com) for the Packrat project (https://rstudio.github.io/packrat/).

The R packages used are listed below:

Package: BH
Source: CRAN
Version: 1.62.0-1
Hash: 14dfb3e8ffe20996118306ff4de1fab2

Package: CDM
Source: CRAN
Version: 5.6-16
Hash: c2a49a54143495291d7542af0cc57a77
Requires: Rcpp, RcppArmadillo, mvtnorm, plyr, polycor, psych, sfsmisc

Package: GPArotation
Source: CRAN
Version: 2014.11-1
Hash: 4821625833bcab5af24257ead9119e01

Package: NMF
Source: CRAN
Version: 0.20.6
Hash: 632aad207ab2e551b33990c9f09ab8af
Requires: RColorBrewer, colorspace, digest, doParallel, foreach,
    ggplot2, gridBase, pkgmaker, registry, reshape2, rngtools, stringr

Package: R6
Source: CRAN
Version: 2.2.2
Hash: b2366cd9d2f3851a5704b4e192b985c2

Package: RColorBrewer
Source: CRAN
Version: 1.1-2
Hash: c0d56cd15034f395874c870141870c25

Package: Rcpp
Source: CRAN
Version: 0.12.11
Hash: 2ee22f3b5e75dec80b1d8eca61cd96d8

Package: RcppArmadillo
Source: CRAN
Version: 0.7.900.2.0
Hash: f574482c4f9ba4abd7bfe068d6739b99
Requires: Rcpp

Package: TAM
Source: CRAN
Version: 1.99993-0
Hash: 5ba96436125988f47ce5cf6d1d8515ee
Requires: CDM, GPArotation, Rcpp, RcppArmadillo, WrightMap, lavaan,
    msm, mvtnorm, plyr, psych, sfsmisc, tensor

Package: WrightMap
Source: CRAN
Version: 1.2.1
Hash: a0c9dd1a7584b7a853338a641bb1a9a0

Package: assertthat
Source: CRAN
Version: 0.2.0
Hash: e8805df54c65ac96d50235c44a82615c

Package: audio
Source: CRAN
Version: 0.1-5
Hash: 9bfdee08c847cc05073bb78abc04b2c8

Package: beepr
Source: CRAN
Version: 1.2
Hash: 01dddbcdecbb5c51ce495fa1d96fe92b
Requires: audio, stringr

Package: bindr
Source: CRAN
Version: 0.1
Hash: e3a02070cf705d3ad1c5af1635a515a3

Package: bindrcpp
Source: CRAN
Version: 0.2
Hash: f5f300f6180e097c7ce174ef3c6307a6
Requires: Rcpp, bindr, plogr

Package: broom
Source: CRAN
Version: 0.4.2
Hash: 7ebcffa46afb467e3f3c5687946f6e1a
Requires: dplyr, plyr, psych, reshape2, stringr, tidyr

Package: coda
Source: CRAN
Version: 0.19-1
Hash: e5aff7792ef292cd4306e58e25e403fe

Package: colorspace
Source: CRAN
Version: 1.3-2
Hash: 0bf8618b585fa98eb23414cd3ab95118

Package: combinat
Source: CRAN
Version: 0.0-8
Hash: 1ddebf0315ec8a5c255adc4da9143b81

Package: curl
Source: CRAN
Version: 2.4
Hash: 6f9efd95fe30fa735c9aa3b4de10b022

Package: dichromat
Source: CRAN
Version: 2.0-0
Hash: 08eed0c80510af29bb15f840ccfe37ce

Package: digest
Source: CRAN
Version: 0.6.12
Hash: e53fb8c58673df868183697e39a6a4d6

Package: doParallel
Source: CRAN
Version: 1.0.10
Hash: df91a7abfa938c06ad87b9a2b9269adb
Requires: foreach, iterators

Package: dplyr
Source: CRAN
Version: 0.7.0
Hash: 010550877f0f439cb57023e32e953380
Requires: BH, R6, Rcpp, assertthat, bindrcpp, glue, magrittr,
    pkgconfig, plogr, rlang, tibble

Package: evaluate
Source: CRAN
Version: 0.10
Hash: c3601a10c987d439e0c63ec635234a76
Requires: stringr

Package: expm
Source: CRAN
Version: 0.999-2
Hash: e8f4ac7e9f15ec8b84b4f7ee5ab3104f

Package: forcats
Source: CRAN
Version: 0.2.0
Hash: e5a3b0b96a39f5581467b0c6366f7408
Requires: magrittr, tibble

Package: foreach
Source: CRAN
Version: 1.4.3
Hash: cd53ef4cf29dc59ce3f8c5c1af735fd1
Requires: iterators

Package: ggplot2
Source: CRAN
Version: 2.2.1
Hash: 46e5cb78836848aa44655e577433f54b
Requires: digest, gtable, lazyeval, plyr, reshape2, scales, tibble

Package: glue
Source: CRAN
Version: 1.1.0
Hash: 60734a419552f14d9c8246307ccb26bd

Package: gridBase
Source: CRAN
Version: 0.4-7
Hash: d4b7f73c0fdf11d18d1e1ae1643ac4ec

Package: gtable
Source: CRAN
Version: 0.2.0
Hash: cd78381a9d3fea966ac39bd0daaf5554

Package: gtools
Source: CRAN
Version: 3.5.0
Hash: 471b2e2452dfb30fdc1dd6f1b567925a

Package: haven
Source: CRAN
Version: 1.0.0
Hash: 90a54c98efa6670de9670ef2a9e70cf6
Requires: BH, Rcpp, hms, readr, tibble

Package: highr
Source: CRAN
Version: 0.6
Hash: aa3d5b7912b5fed4b546ed5cd2a1760b

Package: hms
Source: CRAN
Version: 0.3
Hash: 3fca8a1c97e6cfb297fe3f4690f82c58

Package: httr
Source: CRAN
Version: 1.2.1
Hash: 7de1f8f760441881804af7c1ff324340
Requires: R6, curl, jsonlite, mime, openssl

Package: ic.infer
Source: CRAN
Version: 1.1-5
Hash: a0b79aa2f144b8a8595718d68a4d377a
Requires: kappalab, mvtnorm, quadprog

Package: igraph
Source: CRAN
Version: 1.0.1
Hash: 26ac36402e881905359daabfd9ba4057
Requires: NMF, irlba, magrittr

Package: irlba
Source: CRAN
Version: 2.1.2
Hash: 23fb0f166f3a2bdc2ac7b014b94b57a7

Package: iterators
Source: CRAN
Version: 1.0.8
Hash: 488b93c2a4166db0d15f1e8d882cb1d4

Package: jsonlite
Source: CRAN
Version: 1.3
Hash: b9cc3c97b7a6cd513e9b516fe881d6c9

Package: kappalab
Source: CRAN
Version: 0.4-7
Hash: 1e438c23eff7efa7ca515f288b405c1b
Requires: kernlab, lpSolve, quadprog

Package: kernlab
Source: CRAN
Version: 0.9-25
Hash: bf60122a2e1f073661edb69651a682c2

Package: knitr
Source: CRAN
Version: 1.15.1
Hash: eed7437e2b614c9e3923b7932b16ac97
Requires: digest, evaluate, highr, markdown, stringr, yaml

Package: labeling
Source: CRAN
Version: 0.3
Hash: ecf589b42cd284b03a4beb9665482d3e

Package: lavaan
Source: CRAN
Version: 0.5-23.1097
Hash: 3acbd8e4f2b2ff32317763019c0f3c9f
Requires: mnormt, numDeriv, pbivnorm, quadprog

Package: lavaan.survey
Source: CRAN
Version: 1.1.3.1
Hash: 8d2f91d1003bf16cbd6eeb041628037b
Requires: lavaan, survey

Package: lazyeval
Source: CRAN
Version: 0.2.0
Hash: 3d6e7608e65bbf5cb170dab1e3c9ed8b

Package: lpSolve
Source: CRAN
Version: 5.6.13
Hash: 179cd1170ef9f0a25c689dba0d5b4163

Package: lubridate
Source: CRAN
Version: 1.6.0
Hash: b90f4cbefe0b3c545dd68b22c66a8a12
Requires: stringr

Package: magrittr
Source: CRAN
Version: 1.5
Hash: bdc4d48c3135e8f3b399536ddf160df4

Package: markdown
Source: CRAN
Version: 0.7.7
Hash: fea2343a1119d61b0cc5c0a950d103a3
Requires: mime

Package: mime
Source: CRAN
Version: 0.5
Hash: 463550cf44fb6f0a2359368f42eebe62

Package: mirt
Source: CRAN
Version: 1.24
Hash: 48ad5d8c08758a12a24844fab52fbe35
Requires: GPArotation, Rcpp, RcppArmadillo, numDeriv, sfsmisc

Package: mnormt
Source: CRAN
Version: 1.5-5
Hash: d0d5efbb1fb26d2dc5f9394c223084b5

Package: modelr
Source: CRAN
Version: 0.1.0
Hash: 7c9848bf4d734f38b8ce91022d8de949
Requires: broom, dplyr, lazyeval, magrittr, purrr, tibble, tidyr

Package: msm
Source: CRAN
Version: 1.6.4
Hash: 41df2c2c2204a8f5840deb4ce693dec9
Requires: expm, mvtnorm

Package: munsell
Source: CRAN
Version: 0.4.3
Hash: f96d896947fcaf9b6d0074002e9f4f9d
Requires: colorspace

Package: mvtnorm
Source: CRAN
Version: 1.0-6
Hash: 1f55cdac387721d4eda1126ac9aba73b

Package: numDeriv
Source: CRAN
Version: 2016.8-1
Hash: 3a9d0fc99ba2f6aaa500b3d584962be2

Package: openssl
Source: CRAN
Version: 0.9.6
Hash: 5f4711e142a44655dfea4d64fcf2f641

Package: packrat
Source: CRAN
Version: 0.4.8-1
Hash: 6ad605ba7b4b476d84be6632393f5765

Package: pbivnorm
Source: CRAN
Version: 0.6.0
Hash: 49a3782a63f861575cd5e3ad0559690f

Package: pkgconfig
Source: CRAN
Version: 2.0.1
Hash: 0dda4a2654a22b36a715c2b0b6fbacac

Package: pkgmaker
Source: CRAN
Version: 0.22
Hash: 2e5fc2a6b7eaeb1e1d397a8dc5f54480
Requires: digest, registry, stringr, xtable

Package: plogr
Source: CRAN
Version: 0.1-1
Hash: fb19215402e2d9f1c7f803dcaa806fc2

Package: plyr
Source: CRAN
Version: 1.8.4
Hash: ec0683cf0ab5494db2eff1f31591624e
Requires: Rcpp

Package: polycor
Source: CRAN
Version: 0.7-9
Hash: 2510f040851857643c9a21abf2a30512
Requires: mvtnorm

Package: psych
Source: CRAN
Version: 1.7.5
Hash: 0c076a96de916d0d26d866e83909d961
Requires: mnormt

Package: purrr
Source: CRAN
Version: 0.2.2.2
Hash: faada139260184912fea03f3fea13842
Requires: Rcpp, lazyeval, magrittr, tibble

Package: quadprog
Source: CRAN
Version: 1.5-5
Hash: 69000b7dc41b09977dcf781f909d235c

Package: readr
Source: CRAN
Version: 1.1.0
Hash: e6b4a9edee4410b64e0f528115578a3e
Requires: BH, R6, Rcpp, hms, tibble

Package: readxl
Source: CRAN
Version: 0.1.1
Hash: 6b8ba4ec18d4ac9a505dd799e378eafa
Requires: Rcpp

Package: registry
Source: CRAN
Version: 0.3
Hash: f9447c26b51b8c96f53720c5ff862c93

Package: reshape2
Source: CRAN
Version: 1.4.2
Hash: 01fa9a6b3ead377e4fac84af9f982df9
Requires: Rcpp, plyr, stringr

Package: rlang
Source: CRAN
Version: 0.1.1
Hash: 86c53487ce7f82f0a7cc11c816060910

Package: rngtools
Source: CRAN
Version: 1.2.4
Hash: 4db0661fe95ab6eb3d6339495bf22003
Requires: digest, pkgmaker, stringr

Package: rvest
Source: CRAN
Version: 0.3.2
Hash: c69f7526520bad66fd2111ebe8b1364b
Requires: httr, magrittr, selectr, xml2

Package: scales
Source: CRAN
Version: 0.4.1
Hash: 6368a3249d52d20b366191e9349690b6
Requires: RColorBrewer, Rcpp, dichromat, labeling, munsell, plyr

Package: selectr
Source: CRAN
Version: 0.3-1
Hash: 367275e3dcdd208339e131c7a41bec56
Requires: stringr

Package: sfsmisc
Source: CRAN
Version: 1.1-1
Hash: 00af82c1c08f9a5fb278ca3469b6eaf4

Package: sirt
Source: CRAN
Version: 2.0-25
Hash: eb91764da550eb9e9d510d9af9950f96
Requires: CDM, Rcpp, RcppArmadillo, TAM, coda, combinat, gtools,
    ic.infer, igraph, lavaan, lavaan.survey, mirt, mvtnorm, pbivnorm,
    plyr, psych, sfsmisc, sm, survey

Package: sjmisc
Source: CRAN
Version: 2.4.0
Hash: d59e69f07e440a8ae1a5ff84e759809b
Requires: broom, dplyr, haven, magrittr, psych, purrr, stringdist,
    stringr, tibble, tidyr

Package: sm
Source: CRAN
Version: 2.2-5.4
Hash: ba61d3c567aa65a31df66fcfbb3c5de8

Package: stringdist
Source: CRAN
Version: 0.9.4.4
Hash: ab03f2a40cbc89d6cddf444c93e81a42

Package: stringi
Source: CRAN
Version: 1.1.3
Hash: 458dfc7d682c0f80909521fe482f7c6d

Package: stringr
Source: CRAN
Version: 1.2.0
Hash: 25a86d7f410513ebb7c0bc6a5e16bdc3
Requires: magrittr, stringi

Package: survey
Source: CRAN
Version: 3.31-5
Hash: 96b737ef829cb5e33f85417cc59c0d53

Package: tensor
Source: CRAN
Version: 1.5
Hash: 6a56cfa4bdf989a4b74b7ca542dfaeeb

Package: tibble
Source: CRAN
Version: 1.3.3
Hash: cff4362530fb71e27b84a02054bcb315
Requires: Rcpp, rlang

Package: tidyr
Source: CRAN
Version: 0.6.3
Hash: 6fbf7116f2a9604db53b9c11a44d3cfc
Requires: Rcpp, dplyr, lazyeval, magrittr, stringi, tibble

Package: tidyverse
Source: CRAN
Version: 1.1.1
Hash: 72d5fada870c90b835bbdfc281283c99
Requires: broom, dplyr, forcats, ggplot2, haven, hms, httr, jsonlite,
    lubridate, magrittr, modelr, purrr, readr, readxl, rvest, stringr,
    tibble, tidyr, xml2

Package: xml2
Source: CRAN
Version: 1.1.1
Hash: 35dbee121bb8d76347677290ba1c6a06
Requires: BH, Rcpp

Package: xtable
Source: CRAN
Version: 1.8-2
Hash: 7293235cfcc14cdff1ce7fd1a0212031

Package: yaml
Source: CRAN
Version: 2.1.14
Hash: c81230c3a7d9ba20607ad6b4331173d1








